Putting in draft for now. Feedback and guidance on how to implement proper support for parallel transfer would be greatly appreciated.

Discussed in #14449 (comment)

TODO: Update documentation for iRODS - https://docs.cyberduck.io/protocols/irods/

updating the docs is a separate repo, would have its own PR

Rebased PR on top of master.

Verified the following:

• Parallel/Single stream uploads and downloads works
• Config options defined in new iRODS profile are honored
• PAM authentication works via the pam_password auth scheme
• Other operations work too - rename, delete, etc.

With this PR, users will not be able to calculate checksums on data in an iRODS zone. Cyberduck will report checksums in their hex form if they exist in iRODS. We could add an option to the iRODS profile which allows users to instruct Cyberduck to calculate checksums following an upload, but that didn't feel like the correct approach, mainly because profiles seem to be loaded once on program start and never reread.

As for the unit/integration tests, I'm not sure how the implementation can be tested without a real iRODS server.

Finally, this PR bumps the minimum iRODS version requirement to 4.3.2. A new version of irods4j will be needed before this is merged. See irods/irods4j#126. Will get a new version released this week soon.

@dkocher When I move a file in or out of a directory, I see a new iRODS connection get created. It disappears eventually, but not cleanly.

Can you explain why Cyberduck creates a new connection every time a file is moved?

The move class is very similar to other implementations so it's not clear to me what causes the new connection.

@dkocher When I move a file in or out of a directory, I see a new iRODS connection get created. It disappears eventually, but not cleanly.

Can you explain why Cyberduck creates a new connection every time a file is moved?

The move class is very similar to other implementations so it's not clear to me what causes the new connection.

You will need to return Statefulness.stateless in IRODSProtocol#getStatefulness.

@dkocher When would someone choose stateless or stateful?
If I change the protocol to stateless, how will that affect other operations?
If I continue using stateful, what needs to change to make sure the additional connections are closed immediately?

To provide some context, a single iRODS connection cannot be used to execute multiple API operations in parallel. A single request is sent and the server returns a response.

If you can share credentials with us for your test environment we can run daily integration tests from our CI.

@dkocher When would someone choose stateless or stateful? If I change the protocol to stateless, how will that affect other operations? If I continue using stateful, what needs to change to make sure the additional connections are closed immediately?

To provide some context, a single iRODS connection cannot be used to execute multiple API operations in parallel. A single request is sent and the server returns a response.

We will have to keep it stateful then as otherwise it will be attempted to use a single connection for multiple actions in parallel, i.e. when the user is browsing folders or a file transfer with multiple files in parallel.

I will need to review how we can still support the native copy feature implementation for iRODS that would not require a new connection.

If you can share credentials with us for your test environment we can run daily integration tests from our CI.

We don't have a CI system for people to hook into yet. We have a small set of tools which make it easy to launch one or more iRODS servers for testing. It's likely overkill for your needs though.

With that said, building a containerized environment for testing iRODS is pretty easy. I'm happy to put together a Docker compose project for the iRODS component. That will allow you to launch it on a local computer and have it available for testing.

We will have to keep it stateful then as otherwise it will be attempted to use a single connection for multiple actions in parallel, i.e. when the user is browsing folders or a file transfer with multiple files in parallel.

I will need to review how we can still support the native copy feature implementation for iRODS that would not require a new connection.

So, because the IRODSProtocol class reports iRODS as being stateful, does that mean every operation results in a new iRODS connection being created?

If that's true, why is it that the Move operation results in connections which do not disconnect? No other operation shares that behavior.

I added some log statements to my local build to try and box in what is leading to the additional connections, but it didn't help. However, it did reveal a high number of instantiations of IRODSMoveFeature. Any idea why the move operation is instantiated so much?

We have other usages of Docker Compose containers in integration tests, thus that should be feasible.

Where should I place the Docker Compose project? Is the test directory for irods appropriate?

I figure you can move things around if needed.

@dkocher What triggers the Copy implementation?

Using the Duplicate option within the context menu doesn't appear to trigger it.

Docker compose project added under test directory.

Squashing everything down.

That's surprising. Are you able to launch the docker compose setup from the command line directly?

What OS are you using? What version of docker are you running?

I have edited my comment above. Only reproducible on macOS.

I have edited my comment above. Only reproducible on macOS.

That's expected. iRODS does not support ARM, yet.

One way around this is to launch the docker compose project on a different computer and point the tests at it.

TODO: Update documentation for iRODS - https://docs.cyberduck.io/protocols/irods/

A pull request in the repository would be much appreciated. I am particular interested about compatibility with different iRODS server versions.

No problem. We'll make sure to open a PR to update the docs for iRODS.

I am particular interested about compatibility with different iRODS server versions.

irods4j targets iRODS 4.3.2 and later. That's one of the reasons why the iPlant/CyVerse deployment no longer works with this build. iRODS 4.3.5 is planned as the final release of the iRODS 4.3 series. iRODS 5.0.2 was released at the beginning of October.

irods4j is developed and maintained by the iRODS Consortium and tracks the server's feature set closely. Maintaining compatibility is very important for us and one of the reasons for developing irods4j. Support for iRODS in Cyberduck is guaranteed to be stronger with the switch to irods4j.

I'm especially happy with the introduction of the docker compose project. That will allow you and the rest of the Cyberduck team to improve and fix bugs in the iRODS component without needing to rely on external organizations or even the iRODS team (of course, we're happy to help :-) ).

NTS: Add test showing the PAM issue is resolved.

@dkocher Please see questions at #17341 (comment).

The merge window for 9.3 is closing this week.

That's fine. No need to wait on this PR. It can be part of the next release.

Plus I still need to update the documentation.

NTS: Bump irods4j version to 0.6.0-java8 once it is released.